Regularizing Transformers With Deep Probabilistic Layers

Aurora Cobo Aguilera, Pablo Martinez-Olmos, Antonio Artés-Rodríguez, Fernando Pérez-CruzNeural Networks 2021 · DOI · OpenReview

TL;DR

This paper introduces NORBERT, a novel approach to regularize BERT-like transformer models by integrating a deep generative model (GMVAE) as a stochastic layer. This integration enhances model versatility, improves performance on tasks like missing word imputation, and can lead to more diverse text generation compared to standard BERT.

What Problem Does This Paper Address?

The paper tackles the challenge of improving the robustness and versatility of large language models (LLMs), particularly transformer-based architectures like BERT. While BERT has achieved state-of-the-art results in many Natural Language Understanding (NLU) tasks, it can be prone to overfitting and may lack the flexibility for tasks requiring generative capabilities or nuanced text manipulation. Traditional regularization techniques like dropout, while effective, might not fully address these limitations. The authors aim to introduce a more sophisticated regularization method that leverages the power of deep generative models to inject controlled noise and learn richer representations.

Methods

The core innovation is the integration of a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularization layer within transformer architectures. Instead of using DGMs to directly solve NLU tasks, the GMVAE acts as a stochastic layer that adds controlled noise to the hidden embeddings. This process involves projecting the embeddings into a lower-dimensional latent space and then reconstructing them, introducing a structured form of noise.

The authors propose two main ways to integrate this GMVAE layer into BERT:

  • Top NORBERT: The GMVAE layer is placed on top of the transformer encoder, just before the classification layer. This approach aims to promote topic diversity in generated text.
  • Deep NORBERT: The GMVAE layer is inserted within an intermediate transformer layer (e.g., after self-attention and before the feed-forward block). This approach focuses on improving accuracy and BLEU scores.

The training process involves pre-training BERT, then training the GMVAE on BERT's hidden embeddings, and finally fine-tuning the modified BERT architecture.

Datasets

  • Fashion MNIST (FMNIST): Used for initial experiments to demonstrate the GMVAE's regularization effect on a simpler MLP model.
  • SNLI (Stanford Natural Language Inference): Used for evaluating NORBERT on word imputation tasks.
  • Multi30k: Used for extension experiments with seq2seq models.

Key Insights and Ideas

  • DGMs as Regularizers: The paper demonstrates that DGMs can be effectively used not just for generation but as powerful regularizers within discriminative models like BERT, offering more flexibility than traditional methods.
  • Controlled Noise Injection: The GMVAE's ability to inject structured, non-uniform noise is key. This noise is not random but learned, allowing the model to explore diverse regions of the input space and become more robust.
  • Layer Placement Matters: The location of the GMVAE layer significantly impacts the model's behavior. Placing it at the top (Top NORBERT) enhances diversity, while placing it deeper (Deep NORBERT) improves accuracy and BLEU scores.
  • Beyond Reconstruction: While traditional evaluations focus on reconstructing the original sentence, the authors highlight the potential for DGMs to generate diverse, semantically related, and grammatically correct alternatives, suggesting a new avenue for text generation evaluation.

Key Results

  • Deep NORBERT: Achieved improved accuracy in missing word imputation tasks compared to baseline BERT, with gains of up to 0.2% in some configurations. For instance, 1-Deep NORBERT outperformed BERT by 0.19% on masked tokens and 0.13% on disrupted tokens.
  • Top NORBERT: While showing a significant drop in accuracy (77.14% masked, 75.53% disrupted) compared to Deep NORBERT, Top NORBERT demonstrated a notable increase in topic diversity and generated more varied, semantically related sentences.
  • Seq2seq Extension: The GMVAE regularization also showed promise in improving seq2seq models, particularly in generating more diverse and contextually appropriate reconstructions.
  • Overfitting Mitigation: The GMVAE layer helped prevent overfitting in simpler MLP models, allowing for continued training for many more epochs without performance degradation, unlike standard dropout.

Limitations and Future Work

  • Evaluation Metrics: The authors acknowledge the lack of perfect metrics for evaluating text generation diversity, as standard metrics like BLEU and ROUGE focus on reconstructing the original sentence. They suggest future work on developing metrics that better capture diversity.
  • Computational Cost: Integrating DGMs can increase computational complexity, although the paper doesn't explicitly quantify this.
  • Exploration of Layer Placement: While they explored several layers for Deep NORBERT, a more exhaustive search could potentially yield further improvements.
  • Future Research: The paper suggests exploring new loss functions and metrics for diverse text generation and applying the GMVAE regularization to other transformer architectures and NLU tasks.

Summary updated

Abstract

Language models (LM) have grown with non-stop in the last decade, from sequence-to-sequence architectures to the state-of-the-art and utter attention-based Transformers. In this work, we demonstrate how the inclusion of deep generative models within BERT can bring more versatile models, able to impute missing/noisy words with richer text or even improve BLEU score. More precisely, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer and prove its effectiveness not only in Transformers but also in the most relevant encoder-decoder based LM, seq2seq with and without attention.
BibTeX
@inproceedings{aguilera2021regularizing,
  author = {Aurora Cobo Aguilera and Pablo Martinez-Olmos and Antonio Artés-Rodríguez and Fernando Pérez-Cruz},
  title = {{Regularizing Transformers With Deep Probabilistic Layers}},
  booktitle = {Neural Networks},
  year = {2021},
  doi = {10.1016/j.neunet.2023.01.032},
}
Figure