Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models

Jen-Hao Rick Chang, Ashish Shrivastava, Hema Swetha Koppula, Xiaoshuai Zhang, Oncel Tuzel

Abstract

Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, under an unsupervised-style setting, typical training algorithms for controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but unpaired samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. The proposed method is simple yet effective, where we use a style transformation module to transfer target style information into an unrelated style input. This method enables training using unpaired content and style samples and thereby mitigate the training-inference mismatch. We apply style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. We conduct thorough evaluation, including both quantitative and qualitative user studies. Our results show that by mitigating the training-inference mismatch with the proposed style equalization, we achieve style replication scores comparable to real data in our user studies.

Introduction

Controllable generative models aim to synthesize sequences, such as speech or handwriting, by combining specific content (e.g., text) with a style reference (e.g., a specific voice or penmanship). In an unsupervised setting, explicit labels for style attributes like prosody or accent are unavailable. Consequently, models are typically trained to reconstruct a sample using its own text and its own audio—a "parallel" setting.

Overview of style-content decomposition and the controllable generative model framework.

However, at inference time, the model must operate in a "non-parallel" setting, applying a style from one sample to the content of another. This discrepancy creates a training-inference mismatch. During training, it is often easier for the model to "leak" content through the style encoder rather than learning to synthesize it from the text input. This "content leakage" causes the model to fail during inference, as it attempts to generate the content of the reference clip instead of the provided text.

Methods: Style Equalization

To address this, the authors propose Style Equalization, a training strategy that forces the model to work in a non-parallel setting from the start. The core idea is to simulate the inference environment by mathematically shifting the style of an unrelated sample to match the target.

The Equalization Process

  1. Style Transformation Module ($M$): Instead of training on (Content A, Style A), the model pairs Content A with an unrelated Style B.
  2. Style Difference Estimator ($\phi$): This module calculates a difference vector ($\delta$) between the unrelated Style B and the target Style A.
  3. The Equalization Step: The module $M$ transforms Style B using $\delta$ so that it approximates the style of A. The generative model $G$ then uses this "equalized" style to reconstruct the original sample A.
Comparison between the standard training-inference mismatch and the proposed style equalization method.

To prevent content from sneaking through the style path, the difference estimator employs average pooling. This ensures that only time-invariant style information (like speaker identity) is transferred, while time-dependent content is filtered out. This equalization is applied to 50% of training batches, allowing the model to still learn time-varying style nuances when the content matches.

Experimental Results

The researchers evaluated Style Equalization on speech synthesis (VCTK and LibriTTS datasets) and handwriting synthesis (a large-scale dataset of 600,000 samples).

  • Speech: On the LibriTTS dataset, the proposed method achieved a Word Error Rate (WER) of 7.6%, significantly lower than the 13.4%–30.4% seen in standard Global Style Token (GST) baselines. In user studies, its style replication score (3.5) was identical to that of real human speech.
  • Handwriting: The improvement was even more dramatic. Standard models failed in non-parallel settings with a 69.8% Character Error Rate, while Style Equalization reduced this to 5.5%, producing highly legible and stylistically accurate results.
  • Zero-Shot Capability: The model successfully generalized to speakers and writers it had never seen during training, mimicking their unique characteristics from a single reference sample.

Conclusion

Style Equalization effectively mitigates the training-inference mismatch in unsupervised controllable generative models. By treating style as a manifold where differences can be modeled as vectors, the method prevents content leakage and enables high-fidelity style replication. Future work will focus on hardening style-content separation using alias-free architectures and modeling more complex, time-varying style interactions.

Summary updated

Abstract

Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, under an unsupervised-style setting, typical training algorithms for controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but unpaired samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. The proposed method is simple yet effective, where we use a style transformation module to transfer target style information into an unrelated style input. This method enables training using unpaired content and style samples and thereby mitigate the training-inference mismatch. We apply style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. We conduct thorough evaluation, including both quantitative and qualitative user studies. Our results show that by mitigating the training-inference mismatch with the proposed style equalization, we achieve style replication scores comparable to real data in our user studies.
BibTeX
@inproceedings{chang2022style,
  author = {Jen-Hao Rick Chang and Ashish Shrivastava and Hema Swetha Koppula and Xiaoshuai Zhang and Oncel Tuzel},
  title = {{Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=aMaQjwz5IXI},
}
Figure