A VAE for Transformers with Nonparametric Variational Information Bottleneck

James Henderson, Fabio James FehrICLR 2023 · OpenReview

In the world of deep learning, we often face a trade-off between the representational power of Transformers and the structured, generative latent spaces of Variational Autoencoders (VAEs). While Transformers excel at handling sequences of varying lengths, standard VAEs usually require a fixed-size "bottleneck" vector.

A new paper from ICLR 2023, "A VAE for Transformers with Nonparametric Variational Information Bottleneck," introduces a clever way to marry these two architectures. By treating attention as a probability distribution and using Bayesian nonparametrics, the authors have created a VAE that can dynamically adjust its bottleneck size based on the complexity of the input.

TL;DR

The authors propose the Nonparametric Variational Information Bottleneck (NVIB), a regularizer that allows Transformers to function as VAEs without forcing the input into a fixed-size vector. By modeling latent representations as mixture distributions, the resulting NVAE model can dynamically scale the number of latent vectors it uses, leading to better text generation and smoother sentence interpolation.

What Problem Does This Paper Address?

Standard Variational Information Bottleneck (VIB) methods—the "engine" behind VAEs—are designed for fixed-size vector spaces. However, a Transformer encoder produces a set of vectors (embeddings) that grows linearly with the length of the input text.

To turn a Transformer into a VAE, researchers previously had to either:

  1. Pool everything into one vector: This loses the fine-grained detail that makes Transformers powerful.
  2. Apply VIB to every token: This doesn't allow the model to "compress" the overall information or decide which tokens are actually important.

The challenge was creating a bottleneck that is permutation-invariant (doesn't care about word order) and variable-sized (can use more vectors for complex sentences and fewer for simple ones).

Methods: The Nonparametric VIB

The NVAE model architecture and its NVIB layer, illustrating how attention acts as a denoising mechanism.

The core innovation is the Nonparametric VIB (NVIB). Here is how the authors built it:

  • Attention as Denoising: The authors provide a novel theoretical interpretation of the attention mechanism. They show that "scaled dot-product attention" is mathematically equivalent to Bayesian query denoising, where the encoder's output acts as a prior distribution used to clean up a noisy query from the decoder.
  • Bayesian Nonparametrics: To handle the variable number of vectors, they use Dirichlet Processes (DPs). In statistics, DPs are used to model distributions where the number of components is unknown and potentially infinite.
  • The NVIB Regularizer: They developed a new KL-divergence loss for these Dirichlet Processes. This loss penalizes the model for two things:
  • The amount of information in each individual vector.
  • The total number of vectors used to represent the input.
  • The Reparameterization Trick: Sampling from a Dirichlet Process is usually "discrete" and breaks backpropagation. The authors developed a novel approximation using a blend of Gamma distributions to allow the model to be trained via standard gradient descent.

Key Insights and Ideas

  • Dynamic Latent Size: The model doesn't just learn what to represent; it learns how many vectors it needs. For a simple sentence, the NVIB might prune the representation down to 2-3 effective vectors, while a complex paragraph might retain 20.
  • Smooth Latent Space: Because the latent space is a mixture of distributions rather than a list of coordinates, interpolating between two sentences (e.g., moving from "The cat sat" to "The dog ran") results in much more fluent intermediate sentences compared to standard Transformer baselines.
  • Formalizing Attention: The "Aha!" moment is the formal proof that attention-based representations are essentially mixture distributions. This bridges the gap between the heuristic "scaled dot-product" and formal Bayesian inference.

Key Results

The models were evaluated on two standard English language benchmarks: Wikitext-2 (for proof-of-concept) and Wikitext-103 (over 100 million tokens, to test scaling to complex language modeling).

Performance comparison showing NVAE achieving a competitive trade-off between reconstruction (BLEU) and generation quality (R-PPL).
  • Competitive Performance: The NVAE achieved a better trade-off between reconstruction accuracy (measured by BLEU) and generation quality (measured by Perplexity) than baselines that used fixed-size bottlenecks or simple "dropout" methods.
  • Effective Regularization: The authors demonstrated that by adjusting a single hyperparameter, they could directly control how many latent vectors the model retained.
  • Superior Interpolation: In qualitative tests, NVAE produced "smoother" transitions between sentences. Baselines often "collapsed" during interpolation, producing gibberish or jumping abruptly between the two source sentences.

Limitations and Future Work

While the paper is a significant theoretical step, the authors note a few hurdles:

  • Single-Head Focus: The current mathematical derivation for "test-time denoising attention" is optimized for single-head attention. Extending this to the multi-head attention used in massive models like GPT-4 is "not trivial" and remains future work.
  • Computational Complexity: The nonparametric math is more intensive than standard VAEs, which might make training very large models more expensive.
  • Scaling Up: While they tested on Wikitext-103 (76 million parameters), the next step is seeing how this behaves in the "Billion-parameter" regime of modern Large Language Models (LLMs).

By treating the Transformer's hidden state not as a sequence of numbers, but as a flexible probability distribution, this research opens the door for more "interpretable" and "controllable" generative AI.

Summary updated

Abstract

We propose a Variational AutoEncoder (VAE) for Transformers by developing a Variational Information Bottleneck (VIB) regulariser for Transformer embeddings. We formalise such attention-based representations as mixture distributions, and use Bayesian nonparametrics to develop a Nonparametric VIB (NVIB) for them. The variable number of mixture components supported by nonparametrics captures the variable number of vectors supported by attention, and exchangeable distributions from nonparametrics capture the permutation invariance of attention. Our Transformer VAE (NVAE) uses NVIB to regularise the information passing from the Transformer encoder to the Transformer decoder. Evaluations of a NVAE, trained on natural language text, demonstrate that NVIB can regularise the number of mixture components in the induced embedding whilst maintaining generation quality and reconstruction capacity.
BibTeX
@inproceedings{henderson2023vae,
  author = {James Henderson and Fabio James Fehr},
  title = {{A VAE for Transformers with Nonparametric Variational Information Bottleneck}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=6QkjC_cs03X},
}
Figure