BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks

Jong-Hoon Oh, Ryu Iida, Julien Kloetzer, Kentaro TorisawaACL 2021

TL;DR

This paper introduces BERTAC, a novel architecture that enhances Transformer-based Language Models (TLMs) like BERT by integrating them with a Convolutional Neural Network (CNN) pretrained using a Generative Adversarial Network (GAN)-style approach. This method significantly boosts performance on various NLP tasks, including GLUE benchmarks and open-domain question answering, by providing TLMs with richer, adversarially learned representations.

What Problem Does This Paper Address?

Transformer-based Language Models (TLMs) such as BERT, ALBERT, and GPT-3 have revolutionized Natural Language Processing (NLP) with their impressive performance. However, researchers are constantly seeking ways to further improve these models, either by scaling them up or by developing novel training schemes and architectures. While current TLMs excel at tasks like masked language modeling, there's a desire to explore alternative ways of learning representations that might complement the strengths of Transformers. This paper addresses the question of whether integrating a differently pretrained neural network, specifically a CNN, can enhance the capabilities of existing TLMs.

Methods

The core innovation of BERTAC lies in its two main components:

  • Adversarial Pretraining of a CNN: The authors pretrained a CNN using a GAN-style training scheme on Wikipedia data. Instead of predicting masked tokens (like in standard TLM pretraining), this CNN learns to generate representations of entities from masked sentences. The GAN setup involves a generator (F) that tries to create realistic entity representations from masked sentences, and a discriminator (D) that tries to distinguish these fake representations from real entity representations generated by another network (R). The generator F, once trained, becomes the CNN used in BERTAC. This adversarial process aims to produce more flexible and informative entity representations.
  • Integration with TLMs via TIERs: The pretrained CNN is then integrated into a TLM. The CNN processes a masked version of the input sentence(s) to generate representations. These CNN-derived representations are then fed into custom Transformer layers called "Transformers for Integrating External Representation" (TIERs). TIERs are similar to standard Transformer encoder layers but replace the self-attention mechanism with an attention mechanism that incorporates the CNN's representations. This allows the TLM to leverage the external, adversarially learned information from the CNN.

Datasets

  • Pretraining Data: Wikipedia sentences were used to generate entity-masked sentences for the GAN-style pretraining of the CNN.
  • Evaluation Datasets:
    • GLUE Benchmark: A multi-task benchmark comprising nine tasks, including single-sentence tasks (CoLA, SST-2) and two-sentence tasks (MRPC, QQP, STS-B, MNLI, QNLI, RTE, WNLI).
  • Open-domain QA Datasets: Quasar-T and SearchQA were used for question answering tasks.

Key Insights and Ideas

  • GANs for Text Representation: The paper demonstrates that GANs, traditionally used for image generation, can be effectively adapted for learning rich text representations, specifically for entities, by framing the problem as distinguishing real vs. fake entity representations.
  • Complementary Strengths: The success of BERTAC suggests that CNNs, with their ability to capture local patterns, can offer complementary information to the global context understanding of Transformers, leading to improved overall performance.
  • Task-Independent Pretraining: The GAN-style pretraining of the CNN is task-independent, meaning the pretrained CNN can be readily integrated with various TLMs for diverse downstream tasks without requiring task-specific pretraining.
  • TIERs for Integration: The novel TIER layers provide an effective mechanism for seamlessly integrating external representations (from the CNN) into the Transformer architecture.

Key Results

  • GLUE Performance: BERTAC, when combined with ALBERT, achieved an average performance improvement of 0.7% over the base ALBERT model on the GLUE development set. It also showed comparable performance to state-of-the-art (SOTA) models and achieved SOTA on 3 out of 9 GLUE tasks.
  • Open-domain QA Performance: BERTAC outperformed previous TLM-based and non-TLM-based methods on both Quasar-T and SearchQA datasets, achieving SOTA results on both EM (Exact Match) and F1 scores.
  • Efficiency: The pretraining of the CNN was computationally efficient, and the resulting models could be reused with various TLMs, incurring only a modest increase in parameters (around 14% for ALBERT-xxlarge).

Limitations and Future Work

  • Computational Cost of Pretraining: While the integration is efficient, the initial pretraining of the CNNs, especially for multiple configurations, still requires significant GPU resources (e.g., 180 hours for nine models).
  • Ablation Studies: The paper includes ablation studies that highlight the contributions of each component (GAN-style CNN, TIERs, layer-wise updates), suggesting that all play a crucial role in the performance gains.
  • Future Work: The authors express hope that their work provides a hint for developing novel strong network architectures and training schemes. The modularity of BERTAC suggests it can be extended to incorporate other types of external knowledge or representations.

Summary updated

BibTeX
@inproceedings{oh2021bertac,
  author = {Jong-Hoon Oh and Ryu Iida and Julien Kloetzer and Kentaro Torisawa},
  title = {{BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure