Contrastive Deterministic Autoencoders For Language Modeling

Amur Ghose, Pascal PoupartEMNLP 2023 · OpenReview

Abstract

Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. Training VAEs, especially for text, often runs into the issue of posterior collapse, resulting in loss of representation quality. Deterministic autoencoders avoid this issue, and have been explored particularly well for images. It is however unclear how to best modify a deterministic model designed for images into a successful one for text. We show that with suitable adaptations, we can significantly improve on batch-normed VAEs (BN-VAEs), a strong benchmark for language modeling with VAEs, by replacing them with analogous deterministic models. We employ techniques from contrastive learning to control the entropy of the aggregate posterior of these models to make it Gaussian. The resulting models skip reparametrization steps in VAE modeling and avoid posterior collapse, while outperforming a broad range of VAE models on text generation and downstream tasks from representations. These improvements are shown to be consistent across both LSTM and Transformer-based VAE architectures. Appropriate comparisons to BERT/GPT-2 based results are also included. We also qualitatively examine the latent space through interpolation to supplement the quantitative aspects of the model.

Introduction

In the domain of generative text modeling, Variational Autoencoders (VAEs) are a fundamental tool. However, they are plagued by a phenomenon known as posterior collapse. This occurs when the KL-divergence regularization term dominates the training objective, causing the encoder to ignore input data and simply output the prior distribution (typically a standard normal). Consequently, the decoder stops relying on the latent code, leading to a failure in learning meaningful data representations.

Existing solutions like KL-annealing, word dropout, or batch normalization (BN-VAE) often act as temporary fixes that may compromise reconstruction quality. This paper proposes the Contrastive Deterministic Autoencoder (CEAE), which avoids stochastic sampling entirely during training to bypass the root cause of collapse while maintaining the generative benefits of a structured latent space.

The CEAE Framework

The core innovation of CEAE is the replacement of the standard VAE's stochastic reparameterization trick with a deterministic mapping combined with a contrastive loss acting as an entropic regularizer.

The CEAE architecture features a deterministic flow and a contrastive semi-supervised loss to regularize the latent space.

Deterministic Flow and Contrastive Regularization

Unlike traditional VAEs that sample $z \sim \mathcal{N}(\mu, \sigma^2)$, the CEAE treats the latent code $z$ as a deterministic output ($z = \mu$). To ensure the latent space remains suitable for generation (i.e., follows a Gaussian distribution), the authors utilize a momentum-based contrastive loss. Mathematically, minimizing this contrastive loss is equivalent to maximizing the entropy of the latent space. When combined with batch normalization to maintain zero mean and unit variance, this pushes the aggregate latent distribution toward an isotropic Gaussian.

Experimental Results

The CEAE was evaluated against a wide array of VAE variants using the Yahoo and Yelp corpora. The model demonstrated consistent superiority in reconstruction fidelity and generation quality across different architectures, including LSTMs and Transformers.

Quantitative comparison of CEAE against a broad range of VAE baselines on the Yahoo and Yelp datasets, showing superior reconstruction (Rec) and perplexity metrics.

Key Findings

  • Superior Generation: CEAE consistently outperformed strong baselines like BN-VAE, $\beta$-VAE, and Skip-VAE in reconstruction log-loss and perplexity. By removing the "noise" of sampling, the model achieves higher-fidelity reconstruction.
  • Human and GPT-4 Evaluation: Blind tests revealed a significant preference for CEAE-generated text over traditional VAEs. Evaluators noted improved coherence and grammatical quality.
  • Representation Quality: On downstream sentiment classification tasks, CEAE achieved an impressive 85.6% accuracy with only 100 labeled samples, demonstrating that its deterministic latent codes capture rich semantic information.
  • Architecture Agnostic: The benefits of CEAE were observed both in standard LSTM setups and when integrated into large-scale Transformer frameworks like OPTIMUS (BERT encoder + GPT-2 decoder).

Conclusion

The Contrastive Deterministic Autoencoder provides a robust alternative to VAEs for language modeling. By leveraging contrastive learning as a proxy for entropy maximization, it successfully shapes a deterministic latent space into a Gaussian distribution. This approach effectively eliminates posterior collapse, leading to state-of-the-art performance in both text generation and representation learning. Future work may involve scaling these methods to massive decoder-only architectures and exploring more advanced text-based data augmentations for the contrastive objective.

Summary updated

Abstract

Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. Training VAEs, especially for text, often runs into the issue of posterior collapse, resulting in loss of representation quality. Deterministic autoencoders avoid this issue, and have been explored particularly well for images. It is however unclear how to best modify a deterministic model designed for images into a successful one for text. We show that with suitable adaptations, we can significantly improve on batch-normed VAEs (BN-VAEs), a strong benchmark for language modeling with VAEs, by replacing them with analogous deterministic models. We employ techniques from contrastive learning to control the entropy of the aggregate posterior of these models to make it Gaussian. The resulting models skip reparametrization steps in VAE modeling and avoid posterior collapse, while outperforming a broad range of VAE models on text generation and downstream tasks from representations. These improvements are shown to be consistent across both LSTM and Transformer-based VAE architectures. Appropriate comparisons to BERT/GPT-2 based results are also included. We also qualitatively examine the latent space through interpolation to supplement the quantitative aspects of the model.
BibTeX
@inproceedings{ghose2023contrastive,
  author = {Amur Ghose and Pascal Poupart},
  title = {{Contrastive Deterministic Autoencoders For Language Modeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=TW2cBze4ZB},
}
Figure