Taking Notes on the Fly Helps BERT Pre-training

Qiyu Wu, Chen Xing, Yatao Li, Guolin Ke, Di He, Tie-Yan Liu

Abstract

How to make unsupervised language pre-training more efficient and less resource-intensive is an important research direction in NLP. In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization. It is well-known that in language data corpus, words follow a heavy-tail distribution. A large proportion of words appear only very few times and the embeddings of rare words are usually poorly optimized. We argue that such embeddings carry inadequate semantic signals, which could make the data utilization inefficient and slow down the pre-training of the entire model. To mitigate this problem, we propose Taking Notes on the Fly (TNF), which takes notes for rare words on the fly during pre-training to help the model understand them when they occur next time. Specifically, TNF maintains a note dictionary and saves a rare word's contextual information in it as notes when the rare word occurs in a sentence. When the same rare word occurs again during training, the note information saved beforehand can be employed to enhance the semantics of the current sentence. By doing so, TNF provides better data utilization since cross-sentence information is employed to cover the inadequate semantics caused by rare words in the sentences. We implement TNF on both BERT and ELECTRA to check its efficiency and effectiveness. Experimental results show that TNF's training time is $60\%$ less than its backbone pre-training models when reaching the same performance. When trained with the same number of iterations, TNF outperforms its backbone methods on most of downstream tasks and the average GLUE score.

Introduction

Large-scale language model pre-training, such as BERT and ELECTRA, has revolutionized natural language processing but remains notoriously resource-intensive, often requiring thousands of GPU hours. A major bottleneck in this process is the "heavy-tail" distribution of natural language: while a few words appear constantly, the vast majority appear very rarely.

Because these rare words are seen so infrequently, their embeddings remain poorly optimized and "noisy." This isn't just a local problem; since over 90% of training sequences contain at least one rare word, these inadequate signals act as a drag on the entire model, slowing down the convergence of all parameters. Essentially, the model wastes time trying to make sense of sentences where key information is anchored by a word it barely understands.

Motivation

The proposed method, Taking Notes on the Fly (TNF), is inspired by how humans handle new concepts. When we encounter an unfamiliar term, we often take notes on its context to aid future recall. Similarly, TNF allows a model to store and retrieve contextual "notes" for rare words, effectively creating a form of memory across different training sentences.

Comparison of language understanding with and without contextual notes for rare terms.

As illustrated above, without notes, a rare term like "COVID-19" might be ambiguous to a model in early training. By referencing a note from a previously seen sentence—such as "The COVID-19 pandemic is an ongoing global crisis"—the model can immediately narrow down the term's meaning, leading to more efficient learning.

The TNF Framework

TNF implements this concept through a dynamic Note Dictionary. This dictionary stores historical contextual representations for rare words (defined as those appearing 100–500 times in the corpus).

The TNF architecture, showing the interaction between the Transformer Encoder and the dynamic Note Dictionary.

How TNF Works:

  1. Taking and Updating Notes: When the model encounters a rare word during training, it calculates a "note" by pooling the contextual representations of surrounding words. This new note is merged with the existing entry in the dictionary using an exponential moving average (EMA), allowing the representation to evolve as the model sees the word in different contexts.
  2. Leveraging Notes: When a rare word appears in a training sentence, the model retrieves its current note from the dictionary. This note is weighted-averaged with the word's standard embedding before being fed into the Transformer layers. This provides the model with a semantic "cheat sheet."
  3. Zero-Overhead Deployment: Crucially, the note dictionary is only used during pre-training. For fine-tuning on downstream tasks, the dictionary is discarded. The benefit comes from the Transformer encoder being better trained and more robustly initialized during the initial phase.

Experimental Results

The authors evaluated TNF using the English Wikipedia and BookCorpus for pre-training, with evaluation on the GLUE benchmark.

  • Massive Speedup: TNF reaches the same performance level as its backbone models in 60% less training time. For example, a BERT-base model that normally takes 5.7 days to train can reach equivalent performance in just 2 days with TNF.
  • Superior Performance: When trained for the same number of iterations, TNF consistently outperforms baselines. BERT-TNF achieved an average GLUE score of 83.9 (vs. 83.1 for BERT), while ELECTRA-TNF reached 86.7 (vs. 86.0 for ELECTRA).
  • Small Data Advantage: Performance gains were most prominent on tasks with small datasets (like RTE), suggesting that TNF provides a significantly better initialization point for fine-tuning.

Conclusion

TNF demonstrates that rare words are a systemic bottleneck in language model pre-training. By "fixing" their representations using historical context, the model learns the relationships between all words much faster. This approach shifts the focus from simply training longer to achieving better data utilization, offering a more sustainable path for scaling large-scale models.

Summary updated

Abstract

How to make unsupervised language pre-training more efficient and less resource-intensive is an important research direction in NLP. In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization. It is well-known that in language data corpus, words follow a heavy-tail distribution. A large proportion of words appear only very few times and the embeddings of rare words are usually poorly optimized. We argue that such embeddings carry inadequate semantic signals, which could make the data utilization inefficient and slow down the pre-training of the entire model. To mitigate this problem, we propose Taking Notes on the Fly (TNF), which takes notes for rare words on the fly during pre-training to help the model understand them when they occur next time. Specifically, TNF maintains a note dictionary and saves a rare word's contextual information in it as notes when the rare word occurs in a sentence. When the same rare word occurs again during training, the note information saved beforehand can be employed to enhance the semantics of the current sentence. By doing so, TNF provides better data utilization since cross-sentence information is employed to cover the inadequate semantics caused by rare words in the sentences. We implement TNF on both BERT and ELECTRA to check its efficiency and effectiveness. Experimental results show that TNF's training time is $60\%$ less than its backbone pre-training models when reaching the same performance. When trained with the same number of iterations, TNF outperforms its backbone methods on most of downstream tasks and the average GLUE score. Source code is attached in the supplementary material.
BibTeX
@inproceedings{wu2021taking,
  author = {Qiyu Wu and Chen Xing and Yatao Li and Guolin Ke and Di He and Tie-Yan Liu},
  title = {{Taking Notes on the Fly Helps BERT Pre-training}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=cQcb2X4hQSE},
}
Figure