Unsupervised Controllable Text Generation with Global Variation Discovery and Disentanglement

Peng Xu, Yanshuai Cao, Jackie Chi Kit Cheung

Abstract

The variational autoencoder (VAE) can learn the manifold of natural images on certain datasets, as evidenced by meaningful interpolating or extrapolating in the continuous latent space. However, on discrete data such as text, it is unclear if unsupervised learning can discover similar latent space that allows controllable manipulation. In this work, we find that sequence VAEs trained on text fail to properly decode when the latent codes are manipulated, because the modified codes often land in holes or vacant regions in the aggregated posterior latent space, where the decoding network fails to generalize. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method outperforms unsupervised baselines and strong supervised approaches on text style transfer, and is capable of performing more flexible fine-grained control over text generation than existing methods.

The Latent Vacancy Problem

In computer vision, VAEs are renowned for learning disentangled representations, allowing users to manipulate specific attributes like lighting or pose by adjusting single dimensions in the latent space. However, applying this to text has proven difficult. Standard text VAEs often produce gibberish or ignore style-transfer commands when latent codes are perturbed.

The authors identify the root cause as Latent Vacancy. Unlike images, which occupy a relatively continuous manifold, text data is discrete. This causes the VAE's aggregated posterior—the region of latent space actually covered by training data—to be fragmented, filled with "holes" and disconnected islands. When a latent code is shifted to modify a style (e.g., from positive to negative sentiment), it frequently lands in a "vacuum" where the decoder has no learned mapping, leading to generation failure.

Conceptual illustration of the Latent Vacancy problem where manipulated codes land in regions outside the training distribution.

Methodology: Constrained Posterior VAE (CP-VAE)

To address this, the researchers propose CP-VAE, which "patches" these latent holes through geometric constraints and topological analysis.

Topological Diagnostics

Using Topological Data Analysis (TDA)—specifically the Mapper algorithm—the authors visualized the latent space. They found empirical evidence that text VAEs have significantly more disconnected components than image VAEs, confirming the structural nature of the vacancy problem. CP-VAE aims to restore this connectivity.

Mapper algorithm visualizations showing that text VAEs (middle) have more fragmented latent spaces than image VAEs (left), while CP-VAE (right) restores connectivity.

The Probability Simplex

CP-VAE restricts the "relevant" latent factors ($z^{(1)}$) to a learned probability simplex. By forcing the data onto a bounded geometric shape where corners represent pure attributes (like specific topics or sentiments), the model ensures that any point within the simplex is a valid, decodable representation.

Key components include:

  • Orthogonality Regularization ($L_{REG}$): Forces basis vectors to be orthogonal, ensuring corners represent independent factors.
  • Simplex Filling ($L_{S-REC}$): A margin-based reconstruction loss that pushes training samples to cover the entire simplex surface, eliminating internal holes.
  • Latent Splitting: Separates global controllable factors ($z^{(1)}$) from local stochastic details ($z^{(2)}$) to prevent posterior collapse.

Experimental Results

The model was evaluated on Yelp, Amazon, and AG News benchmarks, focusing on unsupervised discovery of sentiment and topic.

  • Sentiment Transfer: CP-VAE significantly outperformed $\beta$-VAE baselines in content preservation (BLEU scores) while maintaining high classification accuracy for the target style. It achieved results competitive with state-of-the-art supervised models despite being entirely unsupervised.
  • Dynamic Topic Control: CP-VAE demonstrated the ability to switch topics mid-sentence fluently. For instance, it could transition a sentence from a "Sports" context to "Business" by swapping latent codes during the decoding process, maintaining grammatical coherence throughout.

Conclusion

CP-VAE provides a structural solution to the latent vacancy problem in text generation. By constraining the latent space to a well-filled simplex, it enables robust, unsupervised discovery and manipulation of global text attributes. While challenges remain regarding hyperparameter sensitivity and prior matching, this work marks a significant step toward flexible, unsupervised controllable text generation.

Summary updated

Abstract

The variational autoencoder (VAE) can learn the manifold of natural images on certain datasets, as evidenced by meaningful interpolating or extrapolating in the continuous latent space. However, on discrete data such as text, it is unclear if unsupervised learning can discover similar latent space that allows controllable manipulation. In this work, we find that sequence VAEs trained on text fail to properly decode when the latent codes are manipulated, because the modified codes often land in holes or vacant regions in the aggregated posterior latent space, where the decoding network fails to generalize. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method outperforms unsupervised baselines and strong supervised approaches on text style transfer, and is capable of performing more flexible fine-grained control over text generation than existing methods.
BibTeX
@inproceedings{xu2020unsupervised,
  author = {Peng Xu and Yanshuai Cao and Jackie Chi Kit Cheung},
  title = {{Unsupervised Controllable Text Generation with Global Variation Discovery and Disentanglement}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=RALpcVM3Ca},
}
Figure