Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models

Jen-Hao Rick Chang, Ashish Shrivastava, Hema Swetha Koppula, Xiaoshuai Zhang, Oncel Tuzel

TL;DR

Style Equalization is a novel unsupervised training technique that solves the "content leakage" problem in controllable generative models like text-to-speech and handwriting synthesis. By learning to transform the style of one sample to match another during training, the model can be trained on unpaired data. This leads to high-fidelity style replication and significantly better content accuracy without needing any human labels.

The Challenge: Training-Inference Mismatch

When building "controllable" models—like a voice synthesizer that can mimic any speaker—researchers usually face a major hurdle called the training-inference mismatch.

During training, a model typically receives a sample (e.g., an audio clip) and its corresponding text. The model learns to use the audio clip as a "style reference" to reconstruct that exact same clip. Because the style reference contains the same words the model is trying to generate, the model often takes a shortcut: it learns to simply "copy" the audio instead of actually listening to the text input. This phenomenon is known as content leakage.

Training-inference mismatch and the proposed Style Equalization method

At inference time, however, we want the model to take the style of Person A but speak the words of Person B. Because the model learned to "copy" during training, it fails in this "non-parallel" setting, often producing garbled speech or trying to speak the original reference's words instead of the new text. Existing solutions usually involve "bottlenecks" that intentionally cripple the style encoder to prevent it from seeing content, but this sacrifices fine style details like prosody or background noise.

Proposed Method: Style Equalization

To tackle this, the authors introduce Style Equalization, a method that allows models to be trained in the same "non-parallel" way they are used at inference.

  1. Style Transformation Module ($M$): Instead of feeding the ground-truth sample's style directly into the generator, the authors take a different random sample ($x'$) from the dataset.
  2. Difference Estimator ($\phi$): A learnable module calculates the style difference ($\delta$) between this random sample and the target ground-truth sample.
  3. Equalization: The module $M$ transforms the style of the random sample to match the target style.
  4. Training Objective: The generator is then tasked with reconstructing the target sample using the provided text and this "equalized" style from a completely different source.

Crucially, the authors design the style difference $\delta$ to be time-invariant (using average pooling). Since "content" (words/letters) changes rapidly over time while "style" (voice/penmanship) remains relatively constant, this architectural choice ensures that only style information—not words—can pass through the transformation.

Datasets and Evaluation

The researchers evaluated their method across two distinct domains:

  • Speech Synthesis: They used the VCTK dataset (110 speakers, 44 hours) and the much larger LibriTTS dataset (over 2,300 speakers, 555 hours).
  • Handwriting Synthesis: They used a proprietary dataset of 600,000 online handwriting samples from 1,500 writers across multiple languages (English, French, German, etc.).

Key Insights

  • Breaking the Shortcut: By forcing the model to reconstruct Sample A using the style of Sample B (transformed to look like A), the model cannot simply copy the content. It is forced to rely on the text input for content and the style module for the "vibe."
  • Time-Invariance as a Prior: The "aha moment" here is leveraging the temporal nature of sequences. By using global average pooling in the style difference estimator, they create a natural barrier that blocks time-dependent content (phonemes) while letting global style (timbre, microphone response) flow through.
  • Zero Supervision: Unlike many state-of-the-art TTS models that require speaker IDs or pre-trained speaker embeddings, Style Equalization is entirely unsupervised. It learns what "style" is just by looking at the data.

Key Results

The results show that Style Equalization dramatically outperforms standard unsupervised methods like Global Style Tokens (GST):

  • Content Accuracy: In non-parallel speech tasks, the Word Error Rate (WER) dropped from 27.5% (GST-64) to just 9.5% with the proposed method. In handwriting, the Character Error Rate (CER) dropped from 69.8% to 5.5%.
  • Style Fidelity: In user studies, the "Style Opinion Score" for the proposed method (3.5/4.0) was virtually identical to the "Oracle" (real recordings from the same speaker).
  • Generalization: The model successfully replicated the voices of "unseen" speakers (speakers not in the training set) with high accuracy, achieving a cosine similarity of 0.57 compared to 0.07 for baseline unsupervised models.
  • Interpolation: Because the style is represented in a continuous space, the model can smoothly transition between two different voices or handwriting styles by simply sliding a scale between their respective style vectors.

Limitations and Future Work

  • Residual Leakage: The authors admit that while their method reduces content leakage, it doesn't perfectly eliminate it. Because the convolutional layers have a finite receptive field, some local content information might still sneak through.
  • Time-Invariant Assumption: The method assumes style is constant over time. However, some aspects of style, like emotional prosody or specific intonation patterns, are time-dependent. Future work could involve more sophisticated ways to separate time-varying style from time-varying content.
  • Architectural Sensitivity: The success of the method depends on the design of the transformation module $M$; if $M$ is too powerful, it could find new ways to leak content.

Summary updated

Abstract

Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, typical training algorithms for these controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but different samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. By introducing a style transformation module that we call style equalization, we enable training using different content and style samples and thereby mitigate the training- inference mismatch. To demonstrate its generality, we applied style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. Our models achieve state-of-the-art style replication with a similar mean style opinion score as the real data. Moreover, the proposed method enables style interpolation between sequences and generates novel styles.
BibTeX
@inproceedings{chang2022style,
  author = {Jen-Hao Rick Chang and Ashish Shrivastava and Hema Swetha Koppula and Xiaoshuai Zhang and Oncel Tuzel},
  title = {{Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=aMaQjwz5IXI},
}
Figure