On Variational Learning of Controllable Representations for Text without Supervision

Peng Xu, Jackie Chi Kit Cheung, Yanshuai Cao

Abstract: The variational autoencoder (VAE) can learn the manifold of natural images on certain datasets, as evidenced by meaningful interpolating or extrapolating in the continuous latent space. However, on discrete data such as text, it is unclear if unsupervised learning can discover similar latent space that allows controllable manipulation. In this work, we find that sequence VAEs trained on text fail to properly decode when the latent codes are manipulated, because the modified codes often land in holes or vacant regions in the aggregated posterior latent space, where the decoding network fails to generalize. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method outperforms unsupervised baselines and strong supervised approaches on text style transfer, and is capable of performing more flexible fine-grained control over text generation than existing methods.

Introduction

Variational Autoencoders (VAEs) are renowned in computer vision for their smooth latent spaces, allowing users to manipulate attributes like facial expressions by simply moving a point in the latent space. However, when applied to text, these models often break. Manipulating a latent code to change a sentence's sentiment frequently results in gibberish or a total loss of original meaning. This paper identifies the "latent vacancy problem" as the core obstacle and introduces CP-VAE (Constrained Posterior VAE) to bridge this gap.

The Latent Vacancy Problem

The fundamental issue in text-based VAEs is that the aggregated posterior space is often riddled with "holes"—regions where the model has seen no data during training. When we perform attribute manipulation, the resulting latent code often lands in one of these vacant regions. Since the decoder has never learned to map these specific coordinates to meaningful output, it fails to generalize, leading to incoherent text generation.

Conceptual illustration of the latent vacancy problem, where manipulation moves latent codes into 'holes' that the decoder cannot interpret.

CP-VAE: Learning on a Simplex

To solve this, the authors propose constraining the controllable part of the latent representation to a probability simplex. By treating the latent space as a bounded geometric shape rather than an open field, the model can ensure that every point within the simplex corresponds to a recognizable combination of attributes that the decoder can actually process.

Key technical components include:

  • Latent Splitting: Dividing the latent code $z$ into controllable factors (e.g., sentiment) and local reconstruction information.
  • Simplex Mapping: Mapping the posterior mean into a $K$-dimensional simplex where vertices represent discovered latent attributes.
  • Orthogonality and Reconstruction Regularizers: These ensure the simplex vertices remain distinct and that the space within the simplex is fully "filled" with valid representations.

Topological Evidence of Holes

Using Topological Data Analysis (TDA), the researchers provided visual proof of this phenomenon. By visualizing the connectivity of latent codes as graphs, they demonstrated that the latent space of standard text VAEs is far more disconnected than that of image VAEs, explaining why simple interpolation fails for text.

Topological comparison showing the disconnected nature of standard text VAE latent spaces versus the restored connectivity in CP-VAE.

As shown in the visualization, CP-VAE successfully mitigates the latent vacancy problem by creating a more continuous and connected manifold, allowing for smooth transitions and reliable attribute control.

Key Results

The effectiveness of CP-VAE was validated across several benchmarks:

  • Unsupervised Style Transfer: On Yelp and Amazon review datasets, CP-VAE significantly outperformed standard $\beta$-VAEs. It achieved high sentiment transfer strength while maintaining superior content preservation (BLEU scores), rivaling even supervised state-of-the-art models.
  • Topic Modeling and Control: On the AG News dataset, CP-VAE discovered meaningful semantic categories without labels and enabled "topic transitions," where a single sentence could fluently bridge two different subjects.

Conclusion

CP-VAE represents a significant step forward in unsupervised representation learning for discrete data. By identifying the topological deficiencies of standard text VAEs and introducing a constrained simplex structure, the authors have enabled robust, controllable text generation that discovers meaningful attributes entirely from unlabeled data.

Summary updated

Abstract

The variational autoencoder (VAE) can learn the manifold of natural images on certain datasets, as evidenced by meaningful interpolating or extrapolating in the continuous latent space. However, on discrete data such as text, it is unclear if unsupervised learning can discover similar latent space that allows controllable manipulation. In this work, we find that sequence VAEs trained on text fail to properly decode when the latent codes are manipulated, because the modified codes often land in holes or vacant regions in the aggregated posterior latent space, where the decoding network fails to generalize. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method outperforms unsupervised baselines and strong supervised approaches on text style transfer, and is capable of performing more flexible fine-grained control over text generation than existing methods.
BibTeX
@inproceedings{xu2020variational,
  author = {Peng Xu and Jackie Chi Kit Cheung and Yanshuai Cao},
  title = {{On Variational Learning of Controllable Representations for Text without Supervision}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=RALpcVM3Ca},
}
Figure