BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

Abstract: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

2019.

Introduction

BERT (Bidirectional Encoder Representations from Transformers) represents a fundamental shift in Natural Language Processing (NLP). By pre-training deep bidirectional representations from unlabeled text, BERT allows a single model to be fine-tuned for a wide array of tasks, effectively demonstrating that context—specifically bidirectional context—is the key to language understanding.

The Bidirectional Bottleneck

Prior to BERT, state-of-the-art language models were primarily unidirectional. Models such as OpenAI's GPT processed text from left to right, meaning a specific word could only "see" its preceding context. While models like ELMo attempted to address this by concatenating independent left-to-right and right-to-left models, this resulted in a "shallow" bidirectional representation.

The authors argue that this lack of deep bidirectionality is a major bottleneck. For complex tasks like Question Answering (QA) or Named Entity Recognition (NER), understanding the relationship between a word and its entire surrounding sentence is vital. Standard language models cannot simply look at "future" words during training because the model would eventually "see" the target word it is trying to predict, rendering the training process trivial.

Methodology: Pre-training and Fine-tuning

BERT utilizes the Transformer encoder architecture and introduces a two-stage framework: pre-training on unlabeled data and task-specific fine-tuning.

Overall framework of BERT showing pre-training and fine-tuning procedures.

To enable deep bidirectionality without the "leakage" problem, the researchers introduced two novel unsupervised pre-training objectives:

  1. Masked Language Model (MLM): Inspired by the "Cloze" task, BERT randomly masks 15% of input tokens. The model must predict the original token based solely on its context. To bridge the gap between pre-training (where [MASK] tokens exist) and fine-tuning (where they do not), the authors employed an 80/10/10 strategy: 80% of the time the word is replaced with [MASK], 10% with a random word, and 10% it remains unchanged.
  2. Next Sentence Prediction (NSP): To capture inter-sentence relationships, the model is trained to predict whether Sentence B follows Sentence A in the original corpus. This is critical for tasks like Natural Language Inference (NLI).

Architecture Sizes

  • BERT-Base: 12 layers, 768 hidden units, 110M parameters (sized for direct comparison with GPT).
  • BERT-Large: 24 layers, 1024 hidden units, 340M parameters.

Experimental Results

BERT was evaluated on a massive corpus of 3.3 billion words (BooksCorpus and English Wikipedia) and tested across several benchmarks, including GLUE, SQuAD, and SWAG.

GLUE Test results compared to previous state-of-the-art models.

The results were transformative:

  • GLUE Benchmark: BERT achieved an 80.5% score, representing a 7.7% absolute improvement over the previous state-of-the-art.
  • SQuAD v1.1: Reached a 93.2 F1 score, surpassing human-level performance (91.2).
  • Fine-tuning Efficiency: While pre-training is computationally intensive (requiring days on multiple TPUs), fine-tuning for specific tasks is remarkably efficient, often requiring as little as one hour on a single Cloud TPU.

Key Insights and Conclusion

BERT's success proves that models do not need to be regressive to learn language effectively. By fusing left and right context in every layer, BERT creates a unified architecture that can be adapted to almost any NLP task with the addition of a single output layer. Furthermore, the researchers found that scaling to larger models (BERT-Large) consistently yielded gains even on small datasets, suggesting that large-scale pre-training captures general linguistic features that generalize exceptionally well.

Despite its high inference latency and the computational cost of initial training, BERT has set a new standard for how machines understand human language, paving the way for more sophisticated, context-aware AI systems.

Summary updated

BibTeX
@inproceedings{devlin2019bert,
  author = {Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova},
  title = {{BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=SkZmKmWOWH},
}
Figure