Explicit Acoustic Feature Conditioning for Expressive Speech Synthesis

16 concepts12 papers1 proposal

Standard text-to-speech (TTS) models are typically trained to minimize the reconstruction error between a predicted spectrogram and a ground-truth audio clip. While this objective produces clear speech, it inherently suffers from an "averaging" effect: the model converges on the most statistically probable, neutral prosody, stripping away the rich emotional nuances, pitch glides, and dynamic rhythms of natural human speech.

To inject expressivity back into synthetic speech, early architectures relied on reference encoders to extract abstract latent variables or "style embeddings" from a target audio clip, as explored by Pol van Rijn et al. in Exploring Emotional Prototypes in a High Dimensional TTS Latent Space. However, these high-dimensional latent spaces operate as uninterpretable black boxes. Attempting to manipulate them often leads to severe feature entanglement—for instance, shifting a latent vector to increase emotional intensity might inadvertently alter the speaker's identity or distort the pronunciation of specific phonemes.

To overcome these limitations, a prominent research direction focuses on explicit acoustic feature conditioning. Instead of forcing a network to navigate abstract latent spaces, these methods condition the synthesis pipeline on explicit, physically measurable, and human-interpretable acoustic or articulatory parameters.


Grounding Control in Physical Acoustics

Human speech variation is fundamentally driven by a few core acoustic parameters: fundamental frequency ($F_0$ or pitch), energy (volume), duration (speed), and spectral tilt. By extracting these features from real audio during training and feeding them directly into the TTS model as explicit conditioning signals, the network learns a highly disentangled mapping.

[Text Input] ----> [Encoder] ----+----> [Variance Predictors] ----> [Decoder] ----> [Speech]
                                 |             ^
[Explicit Features] -------------+-------------+
(Pitch, Energy, Duration, etc.)

This mechanism works because it provides a direct physical bottleneck. As demonstrated by Tuomo Raitio et al. in Controllable Neural Text-to-Speech Synthesis Using Intuitive Prosodic Features, conditioning a sequence-to-sequence model on normalized pitch, duration, energy, and spectral tilt allows for independent control over each attribute. Because the model explicitly receives the pitch contour, it does not need to infer expressivity from an ambiguous latent vector; it simply learns to reconstruct the phonetic content wrapped around that specific physical contour.

This control can be refined down to the individual syllable or phone. In Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis, Devang S. Ram Mohan et al. model these fundamental acoustic features explicitly at the phone level. This granular conditioning allows for precise, predictable human-in-the-loop adjustments—such as lengthening a single vowel for emphasis or raising the pitch of a question word—without causing unintended side effects in neighboring words or altering the speaker’s voice quality.

Furthermore, grounding the model in explicit traits makes speech synthesis more robust when data is scarce. Pengyu Cheng et al. demonstrate in Speaker Adaption with Intuitive Prosodic Features for Statistical Parametric Speech Synthesis that integrating explicit traits like fundamental frequency, pitch range, and speech rate prevents the model from absorbing environmental or recording noise during voice cloning, ensuring high-fidelity speaker adaptation even with minimal reference data.


Expanding the Control Palette to Articulatory Phonetics

Prosody is not merely a product of pitch and volume; it is also shaped by the physical mechanics of the human vocal tract. Recent work has expanded explicit conditioning beyond basic acoustic contours into the domain of articulatory phonetics.

Instead of treating style as a global variable, the authors of GTR-Voice: Articulatory Phonetics Informed Controllable Expressive Speech Synthesis, Zehua Kcriss Li et al., introduce trainable embedding layers for specific phonetic dimensions—namely Glottalization, Tenseness, and Resonance (GTR)—directly into a feedforward Transformer encoder. By explicitly labeling these articulatory traits in the training data, the model learns to alter the simulated physical properties of the vocal tract, enabling fine-grained stylistic variations such as breathy or tense speech.

This physical grounding also provides a more inclusive framework for modeling speaker identity and gender diversity. Traditional systems often rely on binary speaker classifications or simple pitch-shifting algorithms that sound artificial. To capture authentic vocal diversity, Eva Szekely et al. developed an architecture detailed in An inclusive approach to creating a palette of synthetic voices for gender diversity that incorporates novel speaker embeddings designed to explicitly control acoustic vocal tract length (aVTL) and emergent components. This allows users to make nuanced adjustments to the resonant characteristics of a synthetic voice, decoupling vocal tract geometry from pitch to achieve authentic self-expression.


Forecasting Style Directly from Text

Explicit acoustic conditioning works exceptionally well when a reference audio clip is available to provide the target contours. However, during standard inference, a user only provides text. Expecting a human operator to manually draw pitch and duration curves for every sentence is impractical. To solve this, researchers design auxiliary text-based style predictors that forecast explicit prosodic representations directly from linguistic context.

[Text Input] ----> [Linguistic Encoder (e.g., BERT)] ----> [Style Predictor] ----> [Predicted Acoustic Features] ----> [TTS Core]

The core challenge here is that text is inherently non-deterministic: a single sentence can be spoken in many different emotional styles. To prevent the style predictor from generating a blurred, average prosody, the mapping must be carefully constrained:


Architectural Integration: Where Do the Controls Go?

Once explicit features or style embeddings are obtained, the final piece of the puzzle is determining where and how to inject them into the neural network to ensure they effectively guide the synthesis process.

Variance Predictors

In modern feedforward architectures (such as FastSpeech-style models), explicit style embeddings are injected directly into the variance adaptors. For example, Liisa Rätsep et al. demonstrate in Enabling Conversational Speech Synthesis using Noisy Spontaneous Data that adding style embeddings directly to the inputs of the duration and pitch predictors allows a Transformer-based architecture to successfully separate conversational speech styles from standard read speech, even when training on noisy, spontaneous data.

Multi-Stage Conditioning

Rather than conditioning the model solely at the input or encoder stage, deeper architectures propagate style control across multiple generation phases. In DurIAN-E 2: Duration Informed Attention Network with Adaptive Variational Autoencoder and Adversarial Learning for Expressive Text-to-Speech Synthesis, Dan Su et al. integrate style embeddings at multiple stages of the model, including the normalizing flow layers and the final waveform decoder. This multi-stage conditioning ensures that the stylistic intent influences both global prosodic structure and fine-grained acoustic waveform generation, preventing mismatches between the text encoder and the vocoder.

Generative Flows and Normalization Layers

Advanced generative frameworks use specialized layers to dynamically alter feature maps based on explicit controls. Ajinkya Kulkarni et al. utilize Inverse Autoregressive Flows (IAF) paired with deep metric learning in Transfer Learning of the Expressivity Using FLOW Metric Learning in Multispeaker Text-to-Speech Synthesis to inject expressivity into a latent variable, allowing for independent control over content, speaker identity, and style. Alternatively, frameworks like TSP-TTS employ Style Hierarchical Layer Normalization (SHLN) to scale and shift intermediate representations dynamically, ensuring that the target emotional style is uniformly expressed across all layers of the network.

Key Foundational Work

Controllable Neural Text-to-Speech Synthesis Using Intuitive Prosodic Features (2020) — contributed 2 concept elements to this direction.
Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis (2021) — contributed 2 concept elements to this direction.
Exploring Emotional Prototypes in a High Dimensional TTS Latent Space (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_explicit_acoustic_feature_control_72a4e92f,
  title = {explicit acoustic feature control},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explicit-acoustic-feature-control/rdr_72a4e92f83bd4ccab21387a58982819d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_explicit_acoustic_feature_control_72a4e92f_v1,
  title = {explicit acoustic feature control},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explicit-acoustic-feature-control/rdr_72a4e92f83bd4ccab21387a58982819d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake