When humans speak, we do not just choose discrete categories of expression. We do not speak in a binary state of "happy" or "sad," nor do we switch flatly between "accented" and "native." Instead, human speech is defined by a continuous spectrum of intensity. A question can be softly implied or sharply demanded; an accent can be a subtle undertone or highly pronounced.
For Text-to-Speech (TTS) and Voice Conversion (VC) systems, replicating this nuance requires fine-grained control over speech attributes like emotion, prosody, and intonation. Traditionally, achieving this control required labor-intensive, manually labeled datasets where annotators scored speech intensity on an absolute scale (e.g., rating anger from 1 to 5). However, absolute scaling is notoriously noisy and subjective—what feels like a "4" in loudness or emotion to one annotator might be a "2" to another.
To bypass this bottleneck, a powerful paradigm has emerged: Relative Attribute Ranking. Instead of asking a model to predict an absolute intensity score, models are trained to compare audio samples and determine which one possesses more or less of a given attribute. By shifting the problem from absolute classification to relative ranking, models can discover and map a continuous spectrum of speech intensity entirely without manual labels.
Why Relative Ranking Works: The Intuition
The core insight behind relative attribute ranking is that ordering is easier and more consistent than absolute measurement. While it is difficult to define exactly what an absolute emotion score of "0.75" means, it is highly intuitive to determine that Audio A is more expressive than Audio B.
By optimizing a model to preserve these relative orderings, the network implicitly learns a scalar function $f(x)$ that maps an audio sample $x$ to a continuous, normalized space.
Absolute Labeling (Flawed):
Audio A ───> [Is this a 3.2 or a 3.5 out of 5?] ───> High Variance / Noisy Labels
Relative Ranking (Robust):
Audio A (Expressive) ┐
├───> [Which is more intense?] ───> Audio A > Audio B ───> Smooth Latent Scale
Audio B (Neutral) ┘
During training, a ranking loss (such as a pairwise ranking loss or a triplet loss) forces the model's predicted scalar values to match the known relative order of the inputs. If sample $A$ is known to have a higher intensity than sample $B$, the loss function penalizes the network unless $f(A) > f(B)$. Over time, the model organizes its latent space such that the scalar output corresponds directly to the physical intensity of the attribute.
Constructing Comparison Pairs Without Manual Labels
If the goal is to eliminate manual annotations, how do we know which audio sample has "more" of an attribute during training? Researchers have developed several clever strategies to construct comparison pairs automatically.
1. Neutral Baselines and Triplet Losses
The simplest way to establish a relative gradient is to compare an expressive speech sample against a neutral baseline of the same text or speaker.
- As demonstrated by Lei Xie et al. and Kun Zhou et al., comparing expressive speech fragments directly against neutral ones allows a model to derive continuous emotion intensity scores from unlabelled data.
- To formalize this, Lingxiao Yang et al. utilized a triplet loss strategy. By ranking an emotional sample against a neutral baseline, the model learns to quantify fine-grained variations within a single emotional category, mapping emotional strength to a continuous scale.
2. Self-Supervised Data Augmentation
To learn acoustic attributes like pitch, volume, or rhythm without any text or style labels, models can rank speech against its own digitally altered variants.
- Shijun Wang et al. introduced a self-supervised approach where a model is trained to rank the relative intensity of pitch and volume across augmented speech samples. Because the augmentation process explicitly dictates which sample has higher pitch or volume, the model learns a robust prosody representation without manual labels.
- Similarly, Jianzong Wang et al. leveraged data augmentations like pitch-shifting and time-stretching combined with ranking losses. By comparing original audio with these synthetic variations, the model automatically isolates and disentangles pitch and rhythm from speaker identity, eliminating the need for hand-tuned information bottlenecks.
3. Pseudo-Parallel Corpora and Synthetic References
For complex attributes where simple signal augmentation fails—such as accent strength—researchers use generative models to create reference pairs.
- Rui Liu et al. addressed accent control by using a zero-shot voice synthesis model to generate native-like reference utterances from non-native speech. This creates a pseudo-parallel corpus. A ranking function is then trained on the acoustic features of these pairs to map the utterances to a normalized accent intensity score, allowing users to dial the accent up or down during inference.
4. Mixup Interpolation
To capture the continuous space between different emotion categories, models can enforce ranking constraints on synthetic interpolations.
- Shijun Wang et al. proposed a Mixup-based ranking strategy that accounts for both inter-class (different emotions) and intra-class (varying levels of the same emotion) distances. By ranking interpolated samples, the model shifts the objective from discrete emotion classification to a continuous spectrum, capturing subtle nuances in emotional expression.
Achieving Fine-Grained Disentanglement and Control
Once the ranking function learns to map an attribute to a continuous scalar value, this value becomes an explicit "control knob" within the generative network.
During the synthesis or voice conversion process, this scalar can be manipulated at different granularities. For instance, Yi Lei et al. demonstrated that treating intensity as a relative attribute allows for syllable-level emotional strength control, enabling users to manually adjust the emphasis of specific words by ranking acoustic features like energy and pitch.
This mechanism is highly versatile and extends beyond emotion:
- Intention and Intonation: Haobin Tang et al. applied relative attribute ranking to establish a continuous scale for questioning strength, allowing a model to learn a distinct hierarchy of questioning intensity that listeners can reliably distinguish.
- Text-Driven Prediction: While these intensity knobs can be adjusted manually during inference, they can also be automated. Hyun-Wook Yoon et al. showed that a pre-trained language model can be used to predict these continuous ranking-derived scalar values directly from raw text context, allowing a system to automatically infer how intensely an emotion should be expressed based on the narrative.
By framing intensity as a relative relationship rather than an absolute value, these systems achieve highly natural, expressive, and controllable speech synthesis while entirely sidestepping the need for labor-intensive, manually labeled datasets.
Key Foundational Work
Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis (2020) — contributed 2 concept elements to this direction.
Zero-shot Voice Conversion via Self-supervised Prosody Representation Learning (2021) — contributed 2 concept elements to this direction.